Exclude the MISRA Website from CI-CD link verifier checks #91

Skptak · 2023-11-08T19:57:15Z

No description provided.

Skptak · 2023-11-08T19:58:21Z

This fixes the issue seen in FreeRTOS/FreeRTOS-Kernel#880
Putting the fix here so that all the repos that contain this link will receive this change instead of opening a per repo PR

…sequence. Also trying to fix the issue with trailing comas and slashes being counted as part of the URLs.

…token to the action so that workflows can use the CLI

Skptak · 2023-11-08T22:28:41Z

.github/workflows/pr_checks.yml

@@ -160,7 +160,7 @@ jobs:
              org: AWS,
              branch: main,
              run-link-verifier: true,
-              run-complexity: true,
+              run-complexity: false,


Jobs is undergoing changes still
For now skip running build related tests.

Skptak · 2023-11-08T22:28:53Z

.github/workflows/pr_checks.yml

@@ -210,7 +210,7 @@ jobs:
        with:
          path: repo/${{ matrix.inputs.repository }}
          exclude-dirs: complexity, formatting
-          exclude-urls: https://dummy-url.com/ota.bin
+          exclude-urls: https://dummy-url.com/ota.bin, https://s3.region.amazonaws.com/joe-ota


Jobs has a new URL for its own tests, add to this list.

Skptak · 2023-11-08T22:29:20Z

link-verifier/action.yml

@@ -20,6 +20,7 @@ inputs:
  exclude-urls:
    description: 'Comma separated list of URLS not to check'
    required: false
+    default: https://www.misra.org.uk/misra-c, https://www.misra.org.uk


Placing these here just so that if not adding them explicitly it shows up the in action log

Skptak · 2023-11-08T22:30:20Z

link-verifier/action.yml

+      # now has a CAPTCHA landing page, as such always exclude it from this check.
+      touch allowList.txt
+      echo "https://www.misra.org.uk/misra-c" >> allowList.txt
+      echo "https://www.misra.org.uk" >> allowList.txt


Idea here is to make this fix more "portable" as any repos that are already using the above parameters would need an individual PR

Adding the exclusion of these URLs means that we only need to make this change in this repo, not ever repo

Skptak · 2023-11-08T22:31:08Z

link-verifier/fileTests/goodFiles/fileWithLowercasemdIntheName.md

+# Test that it will find this url and drop the slash
+https://www.google.com/
+# Test that it will find this url by dropping the coma
+https://www.google.com,


Test to make sure we don't try and use the trailing coma as part of the URL, currently an issue with the checker.

Skptak · 2023-11-08T22:31:36Z

link-verifier/verify-links.py

@@ -14,7 +14,7 @@
 import traceback
 from collections import defaultdict

-MARKDOWN_SEARCH_TERM = r'\.md$'
+MARKDOWN_SEARCH_TERM = r"\.md$"


This still throws a warning about it not being a valid escape sequence? But not sure what it wants.

Skptak · 2023-11-08T22:32:14Z

link-verifier/verify-links.py

@@ -14,7 +14,7 @@
 import traceback
 from collections import defaultdict

-MARKDOWN_SEARCH_TERM = r'\.md$'
+MARKDOWN_SEARCH_TERM = r"\.md$"
 # Regex to find a URL
 URL_SEARCH_TERM = r'(\b(https?)://[^\s\)\]\\"<>]+[^\s\)\.\]\\"<>])'


It would make sense to update this regex to exclude the trailing slash, or coma, but I honestly have no idea how this even matches currently.

Skptak · 2023-11-08T22:33:36Z

link-verifier/verify-links.py

@@ -151,7 +151,11 @@ def identify_broken_links(self, files, verbose):
                cprint(f'\t{link}','green')

        for link in self.external_links:
-            is_broken, status_code = test_url(link)
+            # Remove the trailing slash or trailing coma


Trailing slash - So that we don't do a duplicate search of <URL>/ and <URL>

Trailing coma - There are a few places I've seen links be put into files like this
/* <COMMENT> <URL>, <MORE COMMENT> */
Where this then breaks the URL checker, since the current regex grabs the coma

Skptak · 2023-11-08T22:34:36Z

link-verifier/verify-links.py

@@ -166,7 +170,7 @@ def parse_file(html_file):
    return HtmlFile(html_file)

 def html_name_from_markdown(filename):
-    md_pattern = re.compile('\.md', re.IGNORECASE)
+    md_pattern = re.compile("\.md$", re.IGNORECASE)


Not sure why this uses this, compared to the global one at the top of the file, but just tried using quotes to see if that helped with the warning

Added the $ to make sure it only looks for files that fully end in .md

…nk verification

Skptak · 2023-11-08T22:40:18Z

link-verifier/verify-links.py

@@ -254,7 +258,10 @@ def fetch_issues(repo, issue_type, limit):
        if process.returncode == 0:
            key = issue_type + 's'
            for issue in process.stdout.split():
-                main_repo_list[repo][key].add(int(issue))
+                if(issue.isnumeric()):


If for some reason the GitHub issues can't be accessed this will throw an error attempting to convert it to an int

Check if we have an actual number first, if we do not that means there was an error reading the actual Issues from the repo. When this occurs it returns an output of:

Stdout = ['gh:', 'To', 'use', 'GitHub', 'CLI', 'in', 'a', 'GitHub', 'Actions', 'workflow,', 'set', 'the', 'GH_TOKEN', 'environment', 'variable.', 'Example:', 'env:', 'GH_TOKEN:', '${{', 'github.token', '}}']

Skptak · 2023-11-08T22:41:59Z

link-verifier/verify-links.py

@@ -347,7 +353,7 @@ def main():
                if any(file.endswith(file_type) for file_type in args.include_files):
                    f_path = os.path.join(root, file)
                    if args.verbose:
-                        print("Processing File: {}".format(f_path))
+                        print("\nProcessing File: {}".format(f_path))


Add a newline just to help space the log out when doing a verbose run.

Skptak · 2023-11-08T22:43:07Z

link-verifier/verify-links.py

+        if ( ( link[-1] == "/" ) or ( link[-1] == "," ) ):
+            is_broken, status_code = test_url(link[:-1])
+        else:
+            is_broken, status_code = test_url(link)
        if is_broken:
            broken_links.append(link)


Intentionally use the version of the URL with the coma or slash for the error list. This way the exact link can be searched for in the source file easily.

Exclude the MISRA Website from CI-CD link verifier checks

d8787aa

Skptak force-pushed the main branch 2 times, most recently from 32c8041 to 4a98c0d Compare November 8, 2023 20:27

Trying to remove the warning saying that '\.md' is an invalid escape …

6cb367c

…sequence. Also trying to fix the issue with trailing comas and slashes being counted as part of the URLs.

Skptak force-pushed the main branch from 4a98c0d to 6cb367c Compare November 8, 2023 21:14

Remove the trailing slash or comma from the link being checked.

d80ea09

Skptak force-pushed the main branch from a31f5fd to d80ea09 Compare November 8, 2023 22:01

Skptak added 2 commits November 8, 2023 14:08

Need to exclude from multiple places

9ae86da

Add in two more test cases for the link verifier, pass in the github …

167b073

…token to the action so that workflows can use the CLI

Skptak force-pushed the main branch from c85d48e to 167b073 Compare November 8, 2023 22:21

Skptak commented Nov 8, 2023

View reviewed changes

Raise an error if the issue number isn't correctly read during the li…

0c24b51

…nk verification

Skptak commented Nov 8, 2023

View reviewed changes

Skptak mentioned this pull request Nov 8, 2023

Revert Portable/Renesas formatting FreeRTOS/FreeRTOS-Kernel#876

Merged

2 tasks

Skptak requested review from archigup and ericbj29 November 8, 2023 22:46

archigup approved these changes Nov 8, 2023

View reviewed changes

ericbj29 approved these changes Nov 8, 2023

View reviewed changes

Skptak merged commit b2be421 into FreeRTOS:main Nov 8, 2023
50 checks passed

Skptak mentioned this pull request Nov 8, 2023

CI-CD URL Check Change FreeRTOS/FreeRTOS-Kernel#880

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exclude the MISRA Website from CI-CD link verifier checks #91

Exclude the MISRA Website from CI-CD link verifier checks #91

Skptak commented Nov 8, 2023

Skptak commented Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023 •

edited

Loading

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Skptak Nov 8, 2023 •

edited

Loading

Skptak Nov 8, 2023

Skptak Nov 8, 2023

Exclude the MISRA Website from CI-CD link verifier checks #91

Exclude the MISRA Website from CI-CD link verifier checks #91

Conversation

Skptak commented Nov 8, 2023

Skptak commented Nov 8, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skptak Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skptak Nov 8, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Skptak Nov 8, 2023 •

edited

Loading

Skptak Nov 8, 2023 •

edited

Loading